Systems and methods of operation lock management and system catalog overrides in database systems

ABSTRACT

Systems and methods are provided for a shared memory between a database system and a storage catalog server, which stores system catalog overrides and locks for at least a first workflow. The current system catalog overrides include at least a portion of the system catalog overrides and locks that are retrieved by the shared memory from the storage catalog server at a predetermined point in time. The database system performs a check of at least one of index validity and tenant validity for a second workflow or other database operation by checking a state of the corresponding tenant or index in the current system catalog overrides. The system catalog overrides suppress visibility of the respective system catalog changes when the database system determines that at least one of the index and the tenant is invalid based on the current system catalog overrides for the first workflow.

BACKGROUND

Traditional database systems provide a database catalog having metadatain which definitions of database objects such as base tables, views(virtual tables), synonyms, value ranges, indexes, users, and usergroups are stored. The database catalog is typically within the databasesystem. The database system must retrieve information from the databasecatalog to perform database operations. The database catalog may also bestored in a separate catalog server.

Traditional database systems also provide locks on database operations,where a lock is acquired for a portion of an operation, and the lock isdropped when a portion of the operation is complete. That is, locks areacquired, dropped, and reacquired during the course of performing adatabase operation.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the disclosed subject matter, are incorporated in andconstitute a part of this specification. The drawings also illustrateimplementations of the disclosed subject matter and together with thedetailed description serve to explain the principles of implementationsof the disclosed subject matter. No attempt is made to show structuraldetails in more detail than may be necessary for a fundamentalunderstanding of the disclosed subject matter and various ways in whichit may be practiced.

FIGS. 1A1-1A2 show an example method of performing index or tenantvalidity checks for workflow operations by checking a state of thecorresponding tenant or index in a current system catalog overrides of ashared memory according to an implementation of the disclosed subjectmatter.

FIG. 1A3 shows an example method of performing a lock conflict check inconnection with the method shown in FIG. 1A1 according to animplementation of the disclosed subject matter.

FIG. 1B shows optional operations of the method shown in FIG. 1A1-1A3 tomanage workflows and lock management according to an implementation ofthe disclosed subject matter.

FIG. 2 shows example of using a shared memory between a storage catalogserver having a storage catalog, and a database system having aplurality of instances and backends according to implementations of thedisclosed subject matter.

FIG. 3 shows a workflow and lock management example according toimplementations of the disclosed subject matter.

FIG. 4 shows a computer system according to implementations of thedisclosed subject matter.

FIG. 5 shows a computer, servers and shared memory of a database systemaccording to an implementation of the disclosed subject matter.

DETAILED DESCRIPTION

Implementations of the disclosed subject matter includes a shared memorybetween a database system and a storage catalog server. The sharedmemory stores current system catalog overrides for a workflow of adatabase system, where the current system catalog overrides include atleast a portion of the system catalog overrides that are retrieved bythe shared memory from a separate storage catalog server at apredetermined point in time. The storage catalog server includes astorage catalog to record information about the system catalog overridesfor workflow operations of the database system. The database systemperforms index or tenant validity for workflow operations by checking astate of the corresponding tenant or index in the current system catalogoverrides of the shared memory. The system catalog overrides maysuppress visibility of the respective system catalog changes when thedatabase system determines that at least one of the index and the tenantis invalid based on the current system catalog overrides for the firstworkflow.

The database system may perform a lock conflict check for at least oneof an index and tenant for the second workflow or other databaseoperation by checking a state of the current system catalog overrides inthe shared memory. The database system may delay execution of the secondworkflow or the other database operation until the completion of thefirst workflow when the database system determines a lock conflictexists for at least one of the index and the tenant based on the currentlocks for the first workflow.

By using the shared memory, implementations of the disclosed subjectmatter reduce the amount of disruptive communications with the storagecatalog server every time the database system performs an index ortenant validity check. In particular, overrides may reside in the sharedmemory associated with the storage catalog. The index or tenant validitymay be determined based on checking the shared memory, thus eliminatingthe need to communicate with the catalog server. This reducescomputational and communication overhead for the database system, whileallowing operations to be completed before being recognizable bycomponents of the database system.

In implementations of the disclosed subject matter, locks may be used tomanage workflows, and a workflow daemon may manage and/or scheduleworkflow operations for the database system. Workflow operations maymaintain locks to prevent incompatible concurrent operations, and locksmay be tracked for the active workflow operations. Unlike the locks usedin traditional database systems, the locks in the implementations of thedisclose subject matter may be continuously held to prevent invalidinterleaving of workflow operations so that there is no dropping andreacquiring of locks. Rather, lock ownership is transferred betweenworkflow operations via the workflow daemon, which changes the lockownership via the lock manager.

In some implementations, the system catalog overrides may be used tomaintain consistency between a master database system (which may beknown as a primary database) and disaster recovery (DR) database system.The system catalog overrides may suppress elements of the databaseschema until there is consistency between the master database system andthe DR database system for an operation, and then releases the override.That is, the system catalog overrides stored in the shared memory mayprovide consistency between a master database system and DR databasesystems. The master database system and the DR database system have thesame LSM (log-structured merge-tree) data, but may have differentmetadata stores.

In implementations of the disclosed subject matter, the storage catalogmay maintain locks and system catalog overrides on behalf of anexecuting workflow, and may enforce them continuously until the workflowcompletes. The maintaining of the locks and system catalog overrides mayinclude re-establishing locks and overrides for incomplete workflows atstartup of the database system. The locks provide proper ordering ofworkflows relative to other workflows, as well as to other databaseoperations. A workflow may include one or more operations, which may bereferred to as workflow operations.

A workflow or operation that is blocked by a lock waits until the lockmay be acquired. When the lock is transferred, the workflow or operationmay be performed.

In implementations of the disclosed subject matter, the catalogoverrides do not enforce ordering. Rather, the catalog overridessuppress visibility of one or more system catalog changes to otherworkflows and other database operations until the changes are complete.That is, the one or more system catalog changes may not be recognizableto the other workflows and other database operations until the changesare complete. This allows the database system to avoid stalling whileawaiting completion of slow asynchronous operations affecting the systemcatalog. The combination of workflow managed locks and catalog overridesmay provide fault-tolerant coordination of multi-step LSM metadataoperations between the database and its associated storage catalogserver, and/or minimizing lag when replicating these metadataoperations.

FIGS. 1A1-1A3 show an example method 100 for performing index or tenantvalidity for workflow operations by checking a state of thecorresponding tenant or index in a current system catalog overrides of ashared memory according to an implementation of the disclosed subjectmatter.

At operation 110, a database system (e.g., computer 600, centralcomponent 700, and/or second computer 800 shown in FIG. 4, and/or one ormore of database systems 1200 a-d shown in FIG. 5) may store records andexecute one or more database operations. The database operations mayinclude, for example: organization cloning (e.g., to create a copy of atenant via metadata manipulations in the same database instance);instant org sandboxing (e.g., creating a copy of a tenant via metadatamanipulations in a different database instance within the same storagesubstrate for one or more database instances); org sandboxing (e.g.,creating a copy of a tenant in a different database instance, which maybe in a different data center); org migration (e.g., moving a tenant'sdata to a different database instance, which may be in a different datacenter); online index construction (e.g., building an index on arelation without blocking access to the relation); fast failback (e.g.,when recovering from a disaster recovery (DR) failover, where aread-only instance of the DR database system becomes a master for thedatabase; the old master remains unavailable until it can be resyncedwith the new state of the database, only changed data needs to be copiedback to the original database instance); creating DR standbys (e.g.,create a database backup); upgrade (e.g., creating a new instance andtaking ownership of storage catalog metadata and persistence); drop(e.g., truncation of tables and dropping of tables or indexes to sliceout data), or any other suitable database operation.

At operation 120 a storage catalog server (e.g., storage catalog server850 shown in FIG. 4 and described below) may store a storage catalog(e.g., storage catalog 200 shown in FIG. 2 and described below) torecord information about system catalog overrides for active workflowsof the database system. The storage catalog may include log-structuredmerge-tree (LSM) metadata associated with an instance of the databasesystem, which may persist independently of the database system. Theactive workflows may include a first workflow and/or a second workflow.Each workflow may include one or more operations that modify a state ofthe database system, and/or that manipulate the LSM metadata. The systemcatalog may include the internal database catalog metadata and relationsof the database system, and may be persisted through database loggingand recovery systems. The system catalog overrides may include a requestto suppress visibility of a particular system catalog change to preventaccess to partially constructed database objects, including at least oneof the tenants and the indexes.

At operation 130, current system catalog overrides for at least one of atenant and an index of a first workflow operation of the database systemmay be stored in a shared memory (e.g., shared memory 900 shown in FIG.4) communicatively coupled to the database system and the storagecatalog server. The current system catalog overrides may include atleast a portion of the system catalog overrides that are retrieved bythe shared memory (e.g., the shared memory 900) from the storage catalogserver (e.g., the storage catalog server 850) at a predetermined pointin time. In some implementations, the predetermined point in time maywhen the storage catalog server is updated in response to LSM metadatachanges (e.g., an LSM merge operation, a tenant creation operation, orthe like). The predetermined point in time may include when the storagecatalog may be rewritten when a workflow is started (e.g., eitherdirectly by a user performing a database operation, or indirectly byredoing a log record for the beginning of a workflow). The shared memorymay include the current system catalog overrides for at least the firstworkflow.

The operations shown in FIGS. 1A2-1A3 may be performed concurrently withone another after operation 130, or may be performed sequentially (e.g.,where the operations shown in FIG. 1A2 are performed, and then theoperations shown in FIG. 1A3 are performed).

At operation 140 shown in FIG. 1A2, the database system may perform acheck of at least one of index validity and tenant validity for a secondworkflow by checking a state of the current system catalog overrides inthe shared memory. At operation 143, the respective system catalogchanges may be suppressed when the database system determines that atleast one of the index and the tenant is invalid based on the currentsystem catalog overrides for the first workflow.

FIG. 1A3 shows an example method of performing a lock conflict check inconnection with the method shown in FIG. 1A1 according to animplementation of the disclosed subject matter. At operation 145, thedatabase system may perform a lock conflict check for at least one of anindex and tenant for a second workflow or other database operation bychecking a state of the current system catalog overrides in the sharedmemory. At operation 150, the database system may delay execution of thesecond workflow or other the other database operation until thecompletion of the first workflow operation when the database systemdetermines a lock conflict exists for at least one of the index and thetenant based on the current locks for the first workflow.

FIG. 1B shows optional operations of the method 100 shown in FIG. 1A tomanage workflows and lock management according to an implementation ofthe disclosed subject matter. Locks may be acquired in backends andtransferred to the workflow daemon, which may be responsible for holdingthe locks, such as until the workflow completes. The backends may beprocesses executed by computer 600, central component 700, and/or secondcomputer 800 shown in FIG. 4, and/or one or more of database systems1200 a-d shown in FIG. 5. A workflow daemon may be a process that servesas a resource owner, manager, and/or scheduler of workflow operationsfor the database system (e.g., computer 600, central component 700,and/or second computer 800 shown in FIG. 4, and/or one or more ofdatabase systems 1200 a-d shown in FIG. 5). Held locks may be persistedin the storage catalog (e.g., storage catalog 200 shown in FIG. 2, ofthe storage catalog server 850 shown in FIG. 4), to allow them to bereacquired when, for example, the database system restarts.

In a lock transfer process, a donor process may acquire a lock to betransferred. The donor may initiate a transfer in shared memory (e.g.,the shared memory 900 shown in FIG. 4). A donor process may signal arecipient process, and then may enter a sleep mode with a timeout. Therecipient (e.g., the workflow daemon) may perform an operation to searchfor lock transfer requests. The recipient may then install locks andupdate a local state (e.g., to indicate that the locks have beeninstalled). The recipient may signal the donor, which may wake up fromthe sleep mode. A similar process may be used to request the workflowdaemon to drop a lock that it is holding on behalf of a workflow.

As shown in operation 160 of FIG. 1B, a workflow daemon of the databasesystem may sequence a first workflow operation and a second workflowoperation of the first workflow. That is, in this example, the firstworkflow may include the first workflow operation and the secondworkflow operation. At operation 162, the database system may acquireany locks needed by at least the first workflow operation and transfersthem to the workflow daemon. At operation 164, the workflow daemon maymanage locks for tenants or indexes of at least one of the firstworkflow operation and the second workflow operation, where the locksare durably held by the workflow daemon. At operation 166, the workflowdaemon may manage the locks for at least one of an index and tenant forthe first operation during the first workflow operation. At operation168, the workflow daemon may transfer the lock to the second workflowoperation when the first workflow operation is complete. A primary locktable may be stored in the shared memory. The primary lock table maytrack locks held by at least one of the first workflow operation and thesecond workflow operation. The locking operations shown in FIG. 1B anddescribed above may be continuously held to prevent invalid interleavingof workflow operations. Rather than dropping and reacquiring locks as insome traditional systems, lock ownership may be transferred betweenworkflow operations via the workflow daemon, which changes the lockownership via the lock manager.

FIG. 2 shows example of using a shared memory between a storage catalogserver having a storage catalog, and a database system having aplurality of instances and backends according to implementations of thedisclosed subject matter. The backends may be database processes thatacquire resources for a workflow and/or a workflow operation. Thesebackend processes may be executed by computer 600, central component700, and/or second computer 800 shown in FIG. 4, and/or one or more ofdatabase systems 1200 a-d shown in FIG. 5. The storage catalog 200 maybe stored in the storage catalog server 850 shown in FIG. 4 anddescribed below. The storage catalog 200 may include workflow operationidentifiers (e.g., Workflow_Id 1001), which may identify a particularworkflow having one or more workflow operations. Workflows may betransitions between phases of a larger operation that spans multipletransactions in both the database system and the storage catalog (e.g.,storage catalog 200, which may be stored in the storage catalog server850 shown in FIG. 4). The storage catalog contents 200 may includeworkflow operations (e.g., Workflow_Op 1), index overrides (e.g.,Index_Override 1001 8943, Index_Override 1002 8945), and/or tenantoverrides (e.g., Tenant_Override_Count 0, which may indicate that thenumber of tenant overrides currently in the storage catalog is zero).

The shared memory contents 210 may be stored in the shared memory 900shown in FIG. 4 and described below. The shared memory contents 210 mayinclude index overrides and/or tenant overrides for one or more currentworkflows and/or workflow operations. The shared memory contents 210 maybe retrieved from the storage catalog 200 of the catalog server 850 bythe shared memory 900 by at least one predetermined point in time. Insome implementations, the storage catalog 200 may be read by the sharedmemory 900 and stored in the shared memory 900 to form the shared memorycontents 210. Any updates to the shared memory contents 210 based on oneor more workflows and/or workflow operations may be written to theshared memory 900 and the storage catalog 200 of the catalog server 850.As shown in FIG. 2, one or more backends 220 of the database system(e.g., computer 600, central component 700, and/or second computer 800shown in FIG. 4, and/or one or more of database systems 1200 a-d shownin FIG. 5) may be communicatively coupled to the shared memory 900 thatincludes the shared memory contents 210. Backends 200 may be databaseprocesses that acquire resources for a workflow and/or workflowoperation, transfer ownership of them to a workflow daemon, andinitiates the workflow and/or workflow operation.

FIG. 3 shows a workflow and lock management example for a masterdatabase system and a disaster recover (DR) database system according toimplementations of the disclosed subject matter. Most workflows maymaintain locks on relations or on tenant identifiers to preventincompatible concurrent operations. For example, the base relation foran index may be prevented from being changed or dropped while the indexis being constructed. Locks may be used on workflows when moving betweenworkflow states, when the workflow daemon may be managing multipleactive workflows. Locks may be reacquired and/or transferred at startupbefore some incompatible operation could interfere with an activeworkflow.

As shown in FIG. 3, multiple online indexing operations 300 may beperformed on a master database system (e.g., computer 600 and/or centralcomponent 700 shown in FIG. 4; and/or one or more of the databasesystems 1200 a-d shown in FIG. 5) and a disaster recovery (DR) databasesystem (e.g., second computer 800 and/or one or more of the databasesystems 1200 a-d shown in FIG. 5). The DR database system may include aDR standby, which may be a read-only instance managing the DR copy of aproduction database (e.g., the master database system). The DR standbymay receive and apply logs from its associated master, and may be ableto service read-only queries. This shows the same portion of a log,which is generated in real-time on the master database system andreplayed on the DR database system.

Workflows may avoid potential consistency problems and locking conflictson the DR standby of the DR database system by being be constrained tostart in the same order they did on the master database system. As shownin FIG. 3, the master database system indexing operations 302 mayinclude: create index on A operation 306, create index on C operation310, create index on B operation 314, and drop table B operation 320.The DR database system indexing operations 304 may be performed in thesame order as the master database system indexing operations 302, andmay include: create index on A operation 308, create index on Coperation 312, create index on B operation 316, and drop table Boperation 322.

Any locks needed by a workflow for the master database system indexingoperations 302 may be identified, acquired, and communicated via a logso they may be reacquired for the DR database system indexing operations304. In some implementations, a WORKFLOW BEGIN log record operation mayperform this.

As shown in FIG. 3, the drop table B operation 322 of the DR databasesystem indexing operations 304 may be blocked by the relation locks heldby the ongoing create index on B operation 314 workflow of the masterdatabase system indexing operations 302. When redoing the DR databasesystem indexing operations 304 as create index on B operation 316, thelog records may be replayed faster than on the master database system,so there may be situations where a redo may stall (e.g., at redo stalloperation 318 during the create index on B operation 316) until thelocks can be acquired. As the DR database system may have a lessinteractive load than the master database system, it may have moreavailable system resources to devote to log replay than the masterdatabase system, which may allow the DR database system to replay thelog records faster. Implementations of the disclosed subject matter mayprovide a valid ordering of operations when replication redo stalls areperformed as part of the DR database system indexing operations 304.

The workflow subsystem (e.g., the workflow daemon) may track locks onbehalf of active workflows, and may have a persistent location fortracking the active locks. The storage catalog (e.g., storage catalog200 shown in FIG. 2 of the storage catalog server 850 shown in FIG. 4)may be used for tracking system catalog overrides and to persist thelock data. In some implementations, locks may be promoted fromtransaction scope to session scope. That is, the lock may be held untilexplicitly released or the backend holding the lock terminates. Theworkflow daemon may maintain locks on behalf of the workflows itmanages. The workflow daemon may reacquire held locks at startup andrelease them upon conclusion of workflow processing. In someimplementations, locks on a workflow may be released earlier than uponconclusion of the workflow processing. Workflows may involvecoordination of work across multiple transactions. A lock may only beneeded for part of a workflow, such as for a storage space reservationworkflow. The locks may be continuously held to prevent invalidinterleaving of workflow operations which may occur with dropping andreacquiring locks.

In implementations of the disclosed subject matter, locks may beacquired by a workflow in the backend processing of a command thatinitiates the workflow. In this arrangement, stalls and deadlock kills(e.g., premature ending of a transaction) may only affect backendprocesses. These locks may be transferred to the workflow daemon bymodifying their ownership within the lock manager. Successful returnfrom a workflow-initiating function may indicate that the workflow hasbeen initiated with all of its requested resources. The workflow daemonmay hold the locks while operations are performed under the lock.

The management of workflows and locks as discussed above may be used tocreate a snapshot of a tenant of the database system. Creating snapshotsfor use in creating sandboxes may be disclosed, for example, in“FORMATION AND MANIPULATION OF TEST DATA IN A DATABASE SYSTEM,” U.S.Patent Publ. No. 2018/0129585, which is incorporated by reference hereinin its entirety. Creating snapshots may be used in connection withcreating new tenants, such as disclosed in “SYSTEMS AND METHODS OFCREATION AND DELETION OF TENANTS WITHIN A DATABASE,” U.S. patentapplication Ser. No. 15/634,786, filed on Jun. 27, 2017, which isincorporated by reference herein in its entirety. Creating snapshots mayalso be used in connection with database restore operations, such asdisclosed in “SYSTEMS AND METHODS OF RESTORING A DATASET OF A DATABASEFOR A POINT IN TIME,” U.S. patent application Ser. No. 15/634,796, filedon Jun. 27, 2017, which is incorporated by reference herein in itsentirety.

Creating snapshots may include the operations of locking a tenant andlocking an output snapshot to prevent interference from other workflows,waiting for the highest transaction number in persistence to be greaterthan or equal to the requested snapshot point, creating the snapshot,and releasing the locks.

In creating a snapshot, a workflow prologue operation may be executed,where code executing in a backend process of a database system (e.g.,computer 600, central component 700, and/or second computer 800 shown inFIG. 4, and/or one or more of database systems 1200 a-d shown in FIG. 5)may acquire resources needed by the workflow, transfer ownership of theresources to a workflow daemon, and initiate the workflow operations byposting the initial operation. In the workflow prologue operations forcreating snapshots, a tenant identifier may be used to obtain access toa shared lock, and an exclusive lock may be accessed for the snapshotname. Locks may be transferred to the workflow daemon and persisted inthe storage catalog (e.g., storage catalog 200 shown in FIG. 2, which isstored in storage catalog server 850 shown in FIG. 4). The masterdatabase system (e.g., computer 600 and/or central component 700 shownin FIG. 4, and/or one or more of database systems 1200 a-d shown in FIG.5) may post a create snapshot wait operation that includes the tenantidentifier, a transaction number, and the snapshot name.

Upon completion of the prologue operation, in the master database systemand the DR database system (e.g., the second computer 800 shown in FIG.4 and/or one or more of database systems 1200 a-d shown in FIG. 5), await operation (a create snapshot wait) may be performed with respect toa tenant identifier, a transaction number, and the snapshot name. Whenthe highest transaction number in persistence is less than the snapshottransaction number, a sleep operation is performed, and the workflowdaemon may retry later. The master database system may perform acreation of a snapshot operation based on the tenant identifier,snapshot transaction number, and snapshot name.

Performing the create snapshot operation may be based on the tenantidentifier, snapshot transaction number, and snapshot name for themaster database system and the DR database system. An extent referenceset may be created based on the snapshot name for a tenant identifier ata snapshot transaction number. The extent reference set may includelogical references to a physical extent that is stored in physicalstorage (e.g., a storage device), and may be used to virtualize accessto the physical storage. Transaction number filtering may be used sothat no records newer than snapshot transaction number are included inthe snapshot, and extent pruning may be prevented so that no records aremissing from the snapshot. The master database system may post either asuccess operation or a failure operation, depending on whether thesnapshot was successfully created.

The master database system and the DR database system may have theworkflow daemon execute an epilogue operation, which is code that may beexecuted in the workflow daemon that releases resources to be used byworkflow operations, and may arrange for cleanup of a workflow table. Anoperation pointer may be set to zero, a lock release operation may beperformed for the tenant identifier and for the snapshot name, lockpersistence may be removed from the storage catalog, and remainingworkflow metadata removed from storage catalog once a recovery startingposition (RSP) advances. The RSP may be where log replay begins, forexample, for crash recovery. That is, the remaining workflow metadatamay be removed from the storage catalog once the database system hasadvanced to the point where there is no need to redo workflow operationsduring crash recovery.

Using the snapshot described above, a tenant may be created from thesnapshot by locking the snapshot and locking the new tenant to preventinterference from other workflows. Operations for creating the tenantfrom the snapshot may include establishing a system catalog override forthe new tenant, creating the new tenant, dropping the system catalogoverride, and releasing the locks.

In the workflow prologue operations for creating a tenant from asnapshot, the shared locks may be accessed using the snapshot name, andan exclusive lock may be accessed using a new tenant identifier. Asystem catalog override may be installed for the new tenant identifier.Locks and overrides may be transferred to the workflow daemon andpersisted in the storage catalog (e.g., in storage catalog 200 shown inFIG. 2, which is stored at storage catalog server 850 of FIG. 4). Themaster database system may perform a tenant splice using the snapshotname and the new tenant identifier. The tenant splice operation mayinsert the tenant into an LSM of a database instance of the masterdatabase system and/or the DR database system.

Upon completion of the prologue operations, the master database systemand the DR database system may perform a tenant splice operation usingthe snapshot name and the new tenant identifier. The new tenantidentifier may be spliced into the LSM using the snapshot name. Splicingmay be atomic, and failure of a splicing operation may leave the LSMunmodified. The master database system may perform a create tenantoperation using the new tenant identifier when the splicing issuccessful, and may perform an operation when the splicing fails.

In the master database system, the new tenant may be created using thenew tenant identifier. A row may be inserted into a pg_tenant table forthe new tenant identifier, where the pg_tenant table is the systemcatalog relation describing the attributes of the tenant. The masterdatabase may post the success or failure of this new operation, and theworkflow daemon may perform an epilogue operation.

In the epilogue operation, an operation pointer may be set to zero, andlock release operations may be performed by to new tenant identifier andfor the snapshot name. The system catalog override may be removed fornew tenant identifier. Lock and override persistence may be removed fromthe storage catalog, and remaining workflow metadata may be removed fromthe storage catalog when RSP advances.

Implementations of the presently disclosed subject matter may beimplemented in and used with a variety of component and networkarchitectures. FIG. 4 is an example computer 600 suitable forimplementing implementations of the presently disclosed subject matter.As discussed in further detail herein, the computer 600 may be a singlecomputer in a network of multiple computers. As shown in FIG. 4, thecomputer 600 may communicate with a central or distributed component 700(e.g., server, cloud server, database, cluster, application server,etc.). The central component 700 may communicate with one or more othercomputers such as the second computer 800, which may include a storagedevice 810. In some implementations, the central component 700 may be amaster database system, as described above.

The second computer 800 may be a server, cloud server, or the like. Thestorage 810 may use any suitable combination of any suitable volatileand non-volatile physical storage mediums, including, for example, harddisk drives, solid state drives, optical media, flash memory, tapedrives, registers, and random access memory, or the like, or anycombination thereof. In some implementations, the second computer 800may be a DR database system as discussed above.

The storage catalog server 850 may be a server, cloud server, or thelike, and may store and/or provide a storage catalog for the databasesystem (e.g., storage catalog 200 shown in FIG. 2). The storage catalogmay the LSM metadata associated with a database instance, and may bepersisted independently of the actual database. The storage server 850may use and/or include any suitable combination of any suitable volatileand non-volatile physical storage mediums, including, for example, harddisk drives, solid state drives, optical media, flash memory, tapedrives, registers, and random access memory, or the like, or anycombination thereof. The storage catalog server 850 may communicate withone or more other computers such as the computer 600, the centralcomponent 700, the second computer 800, and shared memory 900.

The shared memory 900 may be any suitable combination of any suitablevolatile and non-volatile physical storage mediums, including, forexample, solid-state memory, hard disk drives, solid state drives,optical media, flash memory, tape drives, registers, and random accessmemory, or the like, or any combination thereof. The shared memory maybe communicatively coupled to a processor, memory controller,communications interface, and/or one or more computers. The sharedmemory 900 may store system catalog overrides and/or portions of thesystem catalog that are retrieved from the storage catalog server 850 atone or more predetermined periods of time.

Data may be stored in any suitable format in, for example, the memory670, the fixed storage 630, the central component 700, the storage 810,the storage catalog server 850, and/or the shared memory 900 using anysuitable filesystem or storage scheme or hierarchy. For example, thecentral component 700 and/or the storage 810 may store data using a logstructured merge (LSM) tree with multiple levels. Further, if thesystems shown in FIGS. 4-5 are multitenant systems, the storage may beorganized into separate log structured merge trees for each instance ofa database for a tenant. Alternatively, contents of all records on aparticular server or system may be stored within a single log structuredmerge tree, in which case unique tenant identifiers associated withversions of records may be used to distinguish between data for eachtenant. More recent transactions may be stored at the highest or toplevel of the tree and older transactions may be stored at lower levelsof the tree. Alternatively, the most recent transaction or version foreach record (i.e., contents of each record) may be stored at the highestlevel of the tree and prior versions or prior transactions at lowerlevels of the tree.

The information obtained to and/or from a central component 700 may beisolated for each computer such that computer 600 may not shareinformation with computer 800. Alternatively or in addition, computer600 may communicate directly with the second computer 800.

The computer (e.g., user computer, enterprise computer, etc.) 600 mayinclude a bus 610 which interconnects major components of the computer600, such as a central processor 640, a memory 670 (typically RAM, butwhich may also include ROM, flash RAM, or the like), an input/outputcontroller 680, a user display 620, such as a display or touch screenvia a display adapter, a user input interface 660, which may include oneor more controllers and associated user input or devices such as akeyboard, mouse, WiFi/cellular radios, touchscreen, microphone/speakersand the like, and may be closely coupled to the I/O controller 680,fixed storage 630, such as a hard drive, flash storage, Fibre Channelnetwork, SAN device, SCSI device, and the like, and a removable mediacomponent 650 operative to control and receive an optical disk, flashdrive, and the like.

The bus 610 enable data communication between the central processor 640and the memory 670, which may include read-only memory (ROM) or flashmemory (neither shown), and random access memory (RAM) (not shown), aspreviously noted. The RAM can include the main memory into which theoperating system and application programs are loaded. The ROM or flashmemory can contain, among other code, the Basic Input-Output system(BIOS) which controls basic hardware operation such as the interactionwith peripheral components. Applications resident with the computer 600can be stored on and accessed via a computer readable medium, such as ahard disk drive (e.g., fixed storage 630), an optical drive, floppydisk, or other storage medium 650.

The fixed storage 630 may be integral with the computer 600 or may beseparate and accessed through other interfaces. A network interface 690may provide a direct connection to a remote server via a telephone link,to the Internet via an internet service provider (ISP), or a directconnection to a remote server via a direct network link to the Internetvia a POP (point of presence) or other technique. The network interface690 may provide such connection using wireless techniques, includingdigital cellular telephone connection, Cellular Digital Packet Data(CDPD) connection, digital satellite data connection or the like. Forexample, the network interface 690 may enable the computer tocommunicate with other computers via one or more local, wide-area, orother networks, as shown in FIG. 5.

Many other devices or components (not shown) may be connected in asimilar manner (e.g., data cache systems, application servers,communication network switches, firewall devices, authentication and/orauthorization servers, computer and/or network security systems, and thelike). Conversely, all of the components shown in FIG. 5 need not bepresent to practice the present disclosure. The components can beinterconnected in different ways from that shown. The operation of acomputer such as that shown in FIGS. 4-5 is readily known in the art andis not discussed in detail in this application. Code to implement thepresent disclosure can be stored in computer-readable storage media suchas one or more of the memory 670, fixed storage 630, removable media650, or on a remote storage location.

FIG. 5 shows an example network arrangement according to animplementation of the disclosed subject matter. Four separate databasesystems 1200 a-d at different nodes in the network represented by cloud1202 communicate with each other through networking links 1204 and withusers (not shown). Each of database systems 1200 may be operable to hostmultiple instances of a database, where each instance is accessible onlyto users associated with a particular tenant. Each of the databasesystems may constitute a cluster of computers along with a storage areanetwork (not shown), load balancers and backup servers along withfirewalls, other security systems, and authentication systems. Some ofthe instances at any of systems 1200 may be live or production instancesprocessing and committing transactions received from users or fromcomputing elements (not shown) for ingesting and providing data forstorage in the instances.

One or more of the database systems 1200 a-1200 d may be used as amaster database system, and one or more of the database systems 1200a-1200 d may be used as a DR database system. For example, one or moreauthorized users of the database systems 1200 a-1200 d that areassociated with the original tenant may request that a sandbox becreated by one of the database systems 1200 a-1200 d. The system, forexample 1200c, may include at least one storage device, such as in FIG.6. For example, the storage may include memory 670, fixed storage 630,removable media 650, a storage device included with the centralcomponent 700 and/or the second computer 800, and/or one or more storagedevices associated with one or more of the database systems 1200 a-1200d.

More generally, various implementations of the presently disclosedsubject matter may include or be implemented in the form ofcomputer-implemented processes and apparatuses for practicing thoseprocesses. Implementations also may be implemented in the form of acomputer program product having computer program code containinginstructions implemented in non-transitory and/or tangible media, suchas floppy diskettes, CD-ROMs, hard drives, USB (universal serial bus)drives, or any other machine readable storage medium, wherein, when thecomputer program code is loaded into and executed by a computer, thecomputer becomes an apparatus for practicing implementations of thedisclosed subject matter. Implementations also may be implemented in theform of computer program code, for example, whether stored in a storagemedium, loaded into and/or executed by a computer, or transmitted oversome transmission medium, such as over electrical wiring or cabling,through fiber optics, or via electromagnetic radiation, wherein when thecomputer program code is loaded into and executed by a computer, thecomputer becomes an apparatus for practicing implementations of thedisclosed subject matter. When implemented on a general-purposemicroprocessor, the computer program code segments configure themicroprocessor to create specific logic circuits. In someconfigurations, a set of computer-readable instructions stored on acomputer-readable storage medium may be implemented by a general-purposeprocessor, which may transform the general-purpose processor or a devicecontaining the general-purpose processor into a special-purpose deviceconfigured to implement or carry out the instructions. Implementationsmay be implemented using hardware that may include a processor, such asa general purpose microprocessor and/or an Application SpecificIntegrated Circuit (ASIC) that implements all or part of the techniquesaccording to implementations of the disclosed subject matter in hardwareand/or firmware. The processor may be coupled to memory, such as RAM,ROM, flash memory, a hard disk or any other device capable of storingelectronic information. The memory may store instructions adapted to beexecuted by the processor to perform the techniques according toimplementations of the disclosed subject matter.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific implementations. However, theillustrative discussions above are not intended to be exhaustive or tolimit implementations of the disclosed subject matter to the preciseforms disclosed. Many modifications and variations are possible in viewof the above teachings. The implementations were chosen and described inorder to explain the principles of implementations of the disclosedsubject matter and their practical applications, to thereby enableothers skilled in the art to utilize those implementations as well asvarious implementations with various modifications as may be suited tothe particular use contemplated.

The invention claimed is:
 1. A system comprising: a database system tostore records and execute one or more database operations; a storagecatalog server that includes a storage catalog to record informationabout system catalog overrides for active workflows of the databasesystem; a shared memory, communicatively coupled to the database systemand the storage catalog server, to store current system catalogoverrides and locks for at least one of a tenant and an index of a firstworkflow of the database system, wherein the current system catalogoverrides include at least a portion of the system catalog overrides andlocks that are retrieved by the shared memory from the storage catalogserver at a predetermined point in time, and wherein the shared memoryincludes the current system catalog overrides for at least the firstworkflow, wherein the database system performs a check of at least oneof index validity and tenant validity for a second workflow or otherdatabase operation by checking a state of the current system catalogoverrides in the shared memory, and wherein the system catalog overridessuppress visibility of the respective system catalog changes when thedatabase system determines that at least one of the index and the tenantis invalid based on the current system catalog overrides for the firstworkflow.
 2. The system of claim 1, wherein the storage catalog includeslog-structured merge-tree (LSM) metadata associated with an instance ofthe database system, which persists independently of the databasesystem.
 3. The system of claim 2, wherein at least one of the firstworkflow and the second workflow are selected from the group consistingof: a workflow that includes operations that modify a state of thedatabase system, and a workflow that includes operations that manipulatethe LSM metadata.
 4. The system of claim 1, wherein the system catalogincludes the internal database catalog metadata and relations of thedatabase system, and is persisted through database logging and recoverysystems.
 5. The system of claim 1, wherein the database system performsa lock conflict check for at least one of an index and tenant for thesecond workflow or other database operation by checking a state of thecurrent system catalog overrides in the shared memory, and wherein thedatabase system delays execution of the second workflow or other theother database operation until the completion of the first workflow whenthe database system determines a lock conflict exists for at least oneof the index and the tenant based on the current locks for the firstworkflow.
 6. The system of claim 1, wherein a workflow daemon of thedatabase system sequences a first workflow operation and a secondworkflow operation of the first workflow.
 7. The system of claim 6,wherein the database system acquires any locks needed by at least thefirst workflow operation and transfers them to the workflow daemon. 8.The system of claim 6, wherein the workflow daemon manages locks fortenants of at least one of the first workflow operation and the secondworkflow operation, wherein the locks are durably held by the workflowdaemon.
 9. The system of claim 6, wherein the locks are managed by theworkflow daemon for at least one of an index and tenant for the firstworkflow operation during the first workflow.
 10. The system of claim 6,wherein the lock is transferred by the workflow daemon of the databasesystem to the second workflow operation when the first workflowoperation is complete.
 11. The system of claim 1, wherein a primary locktable is stored in the shared memory, wherein the primary lock tablethat tracks locks held by at least one of the first workflow and thesecond workflow.
 12. A method comprising: storing records and executingone or more database operations with a database system; storing, at astorage catalog server, a storage catalog to record information aboutsystem catalog overrides for active workflows of the database system;storing, in a shared memory communicatively coupled to the databasesystem and the storage catalog server, current system catalog overridesand locks for at least one of a tenant and an index of a first workflowof the database system, wherein the current system catalog overridesinclude at least a portion of the system catalog overrides and locksthat are retrieved by the shared memory from the storage catalog serverat a predetermined point in time, and wherein the shared memory includesthe current system catalog overrides for at least the first workflow;performing, at the database system, a check of at least one of indexvalidity and tenant validity for a second workflow or other databaseoperation by checking a state of the current system catalog overrides inthe shared memory; and suppressing visibility of the respective systemcatalog changes when the database system determines that at least one ofthe index and the tenant is invalid based on the current system catalogoverrides for the first workflow.
 13. The method of claim 12, whereinthe storage catalog includes log-structured merge-tree (LSM) metadataassociated with an instance of the database system, which persistsindependently of the database system.
 14. The method of claim 13,wherein at least one of the first workflow and the second workflow areselected from the group consisting of: a workflow that includesoperations that modify a state of the database system, and a workflowthat includes operations that manipulate the LSM metadata.
 15. Themethod of claim 12, wherein the system catalog includes the internaldatabase catalog metadata and relations of the database system, and ispersisted through database logging and recovery systems.
 16. The methodof claim 12, further comprising: performing, at the database system, alock conflict check for at least one of an index and tenant for thesecond workflow or other database operation by checking a state of thecurrent system catalog overrides in the shared memory; and delayingexecution of the second workflow or other the other database operationuntil the completion of the first workflow when the database systemdetermines a lock conflict exists for at least one of the index and thetenant based on the current locks for the first workflow.
 17. The methodof claim 12, further comprising: sequencing, at a workflow daemon of thedatabase system, a first workflow operation and a second workflowoperation of the first workflow.
 18. The method of claim 17, furthercomprising: acquiring, at the database system, any locks needed by atleast the first workflow operation and transfers them to the workflowdaemon.
 19. The method of claim 17, further comprising: managing, at theworkflow daemon, locks for tenants of at least one of the first workflowoperation and the second workflow operation, wherein the locks aredurably held by the workflow daemon.
 20. The method of claim 17, furthercomprising: managing, at the workflow daemon, the locks for at least oneof an index and tenant for the first workflow operation during the firstworkflow.
 21. The method of claim 17, further comprising: transferring,at the workflow daemon, the lock to the second workflow operation whenthe first workflow operation is complete.
 22. The method of claim 12,wherein a primary lock table is stored in the shared memory, wherein theprimary lock table that tracks locks held by at least one of the firstworkflow and the second workflow.